One focus detection method performed by an image pickup apparatus is an imaging plane phase-difference type method that detects a focus as a phase-difference type using focus detecting pixels formed in an imaging element. The image pickup apparatus disclosed in Patent Literature 1 uses a single microlens and a two-dimensional imaging element that forms a photoelectric conversion unit divided into a plurality of parts, with respect to one pixel. The photoelectric conversion unit divided into the plurality of parts is configured to receive light in areas with different exit pupils of a photographing lens though the single microlens, and performs pupil division. The focus detection is performed as the imaging plane phase-difference type by generating a viewpoint signal derived from the light respectively received by the photoelectric conversion unit divided into the plurality of parts, and calculating an image shift amount from a parallax between a plurality of viewpoint signals, and converting the result to a defocus amount. Patent Literature 2 discloses a method for generating an imaging signal by summing a plurality of viewpoint signals received by a photoelectric conversion unit divided into a plurality of parts.
The plurality of viewpoint signals with respect to the photographed image is equivalent to Light Field data that is information about the spatial distribution and the angle distribution of light intensity. Non-patent Literature 1 discloses a refocus technique that synthesizes an image in a virtual imaging surface different from a imaging plane with the acquired Light Field data, and alters an in-focus position of an image after photographing.